2002.06 July

Data management technology is on the move, fueled by trends both in grassroots popular computing and in high-end research groups. Where we used to think of information technology (IT) systems centered around data bases, there is an increasing focus on data flow: the complex confluence of data from multiple sources – both stored and streaming – with real-time access for a global community of users. These trends have interesting implications for IT in general and health IT in particular. DATA, NETWORKS, AND DISTRIBUTED QUERIES For the last quarter century, institutional IT systems have revolved around databases — stable, reliable, centralized repositories of information. Patient records, billing systems, and online medical libraries are all examples of the centrality of databases in health IT. Relational database technology was developed by researchers in the 1970s and commercialized in the 1980s. In about the same timeframe, a somewhat different population of researchers was connecting and loosely affiliating disparate computers across the globe; in a federation of systems we now call the Internet. These two research communities represent very different computing cultures. Early database researchers focused on “transactional guarantees” and “strict relational semantics”, with the goal of presenting a reliable, contractually valid model of computing over mission-critical data. By contrast, the Internet designers focused on “rough consensus and running code”, with the goal of building a simple, decentralized system that worked pretty well and would be flexible enough to span the globe. These two viewpoints began to merge in distributed database research in the 1980s and more dramatically in the recent popular rise of document retrieval in the World Wide Web. In both cases, distributed query software was proposed to make multiple distributed databases look like one database; in the relational space this is referred to as “data integration” or “content integration” [4, 10], and in web-style document retrieval this is typically called “metasearch” [2]. The vision of distributed query systems is simple: to present users with the illusion of a single unified database, while in fact integrating data on the fly from multiple databases across the globe. In these systems, each user query is translated into a family of queries on multiple databases, with the results consolidated and presented back to the user. To date, efforts at distributed query systems have proven unsuccessful, at least in the commercial sphere. In both document retrieval and relational data, distributed query solutions have been overshadowed by massive centralization efforts: solutions that fetch all the data from multiple systems to one site and build an integrated copy of the databases there. In document retrieval on the web, this is achieved by web crawlers [5] and in relational databases by data warehousing tools [1]. It has seemed, to date, that the technical and political ability to centralize data into traditional IT architectures has outweighed the promise and capabilities of decentralized query solutions. One reason for this is that the distributed query tools have been relatively evolutionary: they attempt to solve the distributed query problem more efficiently than the standard centralized solution, but they do not provide significant new functionality to the end user. Since solutions for centralizing data have also evolved in the last decade, the distributed query solutions have not solved a burning problem for any customer. ‘DATABASE’ CONTINUED ON PAGE 3 Data Capture: Critical Step in Health IT